Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[clickhouse] Internal monitoring endpoint #7114

Merged
merged 18 commits into from
Nov 26, 2024
Merged

[clickhouse] Internal monitoring endpoint #7114

merged 18 commits into from
Nov 26, 2024

Conversation

karencfv
Copy link
Contributor

@karencfv karencfv commented Nov 20, 2024 •
edited
Loading

Overview

This commit adds an endpoint to retrieve timeseries from the system database. For the time being we will only add support for the metric_log and asynchronous_metric_log tables.

This endpoint is still a bit bare bones, but will be a good start to begin monitoring the ClickHouse servers.

Examples

Queries per second

$ curl "http://[::1]:8888/timeseries/metric_log/ProfileEvent_Query/avg" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2883  100  2883    0     0   5912      0 --:--:-- --:--:-- --:--:--  5907
[
  {
    "time": "2024-11-20T04:09:00Z",
    "value": 0.0
  },
  {
    "time": "2024-11-20T04:10:00Z",
    "value": 0.0
  },
  {
    "time": "2024-11-20T04:11:00Z",
    "value": 0.06666666666666667
  }
]

Disk usage

$ curl "http://[::1]:8888/timeseries/asynchronous_metric_log/DiskUsed_default/avg?interval=120&time_range=3600&timestamp_format=unix_epoch" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1786  100  1786    0     0   3728      0 --:--:-- --:--:-- --:--:--  3728
[
 {
    "time": "1732513320",
    "value": 120491427254.85716
  },
  {
    "time": "1732513440",
    "value": 120382774033.06668
  },
  {
    "time": "1732513560",
    "value": 120364752622.93332
  }
]

Related: #6953

@karencfv
Copy link
Contributor Author

Have been playing around with this. I think we can get some sort of Ok looking charts with this data:

Screenshot 2024-11-22 at 2 55 30 PM

@karencfv karencfv changed the title [clickhouse] WIP: Internal monitoring endpoints [clickhouse] Internal monitoring endpoint Nov 25, 2024
@karencfv karencfv marked this pull request as ready for review November 25, 2024 06:11
@karencfv karencfv requested a review from bnaecker November 25, 2024 06:12
andrewjstone
andrewjstone approved these changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@andrewjstone andrewjstone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

clickhouse-admin/api/src/lib.rs Outdated
@@ -136,4 +150,17 @@ pub trait ClickhouseAdminSingleApi {
async fn init_db(
rqctx: RequestContext<Self::Context>,
) -> Result<HttpResponseUpdatedNoContent, HttpError>;

/// Retrieve time series from the system database. The value the average of all
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second sentence appears to be typo'd. Not sure what you are trying to say.

Also, I'd move that sentence after a blank line, so that the doc comments have a short first line. Same for the method above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol, idk what was happening there 😄

clickhouse-admin/src/clickhouse_cli.rs Outdated
let log = self.log.clone().unwrap();
let query = settings.query_avg();

info!(&log, "Querying system database"; "query" => &query);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be an info log? What about a debug or trace log instead?

@karencfv karencfv enabled auto-merge (squash) November 25, 2024 23:07
@karencfv karencfv mentioned this pull request Nov 25, 2024
Open
8 tasks
@karencfv karencfv merged commit 1f83f07 into oxidecomputer:main Nov 26, 2024
19 checks passed
@karencfv karencfv deleted the monitoring-endpoints branch November 26, 2024 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewjstone andrewjstone andrewjstone approved these changes

@bnaecker bnaecker Awaiting requested review from bnaecker

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@karencfv @andrewjstone